Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

Dense Multi-GPU systems have recently gained a lot of attention in the HPC arena. Traditionally, MPI runtimes have been primarily designed for clusters with a large number of nodes. However, with the advent of MPI+CUDA applications and CUDA-Aware MPI runtimes like MVAPICH2 and OpenMPI, it has become important to address efficient communication schemes for such dense Multi-GPU nodes. This coupled with new application workloads brought forward by Deep Learning frameworks like Caffe and Microsoft CNTK pose additional design constraints due to very large message communication of GPU buffers during the training phase. In this context, special-purpose libraries like NVIDIA NCCL have been proposed for GPU-based collective communication on dense GPU systems. In this paper, we propose a pipelined chain (ring) design for the MPI_Bcast collective operation along with an enhanced collective tuning framework in MVAPICH2-GDR that enables efficient intra-/inter-node multi-GPU communication. We present an in-depth performance landscape for the proposed MPI_Bcast schemes along with a comparative analysis of NVIDIA NCCL Broadcast and NCCL-based MPI_Bcast. The proposed designs for MVAPICH2-GDR enable up to 14X and 16.6X improvement, compared to NCCL-based solutions, for intra- and inter-node broadcast latency, respectively. In addition, the proposed designs provide up to 7% improvement over NCCL-based solutions for data parallel training of the VGG network on 128 GPUs using Microsoft CNTK.Comment: 8 pages, 3 figure

Hot-Spot Avoidance With Multi-Pathing Over Infiniband: An MPI Perspective

Large scale InfiniBand clusters are becoming increasingly popular, as reflected by the TOP 500 Supercomputer rankings. At the same time, fat tree has become a popular interconnection topology for these clusters, since it allows multiple paths to be available in between a pair of nodes. However, even with fat tree, hot-spots may occur in the network depending upon the route configuration between end nodes and communication pattern(s) in the application. To make matters worse, the deterministic routing nature of InfiniBand limits the application from effective use of multiple paths transparently and avoid the hot-spots in the network. Simulation based studies for switches and adapters to implement congestion control have been proposed in the literature. However, these studies have focused on providing congestion control for the communication path, and not on utilizing multiple paths in the network for hot-spot avoidance. In this paper, we design an MPI functionality, which provides hot-spot avoidance for different communications, without a priori knowledge of the pattern. We leverage LMC (LID Mask Count) mechanism of InfiniBand to create multiple paths in the network and present the design issues (scheduling policies, selecting number of paths, scalability aspects) of our design. We implement our design and evaluate it with Pallas collective communication and MPI applications. On an InfiniBand cluster with 48 processes, collective operations like MPI All-to-all Personalized and MPI Reduce Scatter show an improvement of 27% and 19% respectively. Our evaluation with MPI applications like NAS Parallel Benchmarks and PSTSWM on 64 processes shows significant improvement in execution time with this functionality

Tissue-Specific Transcriptomics of the Exotic Invasive Insect Pest Emerald Ash Borer (Agrilus planipennis)

BACKGROUND: The insect midgut and fat body represent major tissue interfaces that deal with several important physiological functions including digestion, detoxification and immune response. The emerald ash borer (Agrilus planipennis), is an exotic invasive insect pest that has killed millions of ash trees (Fraxinus spp.) primarily in the Midwestern United States and Ontario, Canada. However, despite its high impact status little knowledge exists for A. planipennis at the molecular level. METHODOLOGY AND PRINCIPAL FINDINGS: Newer-generation Roche-454 pyrosequencing was used to obtain 126,185 reads for the midgut and 240,848 reads for the fat body, which were assembled into 25,173 and 37,661 high quality expressed sequence tags (ESTs) for the midgut and the fat body of A. planipennis larvae, respectively. Among these ESTs, 36% of the midgut and 38% of the fat body sequences showed similarity to proteins in the GenBank nr database. A high number of the midgut sequences contained chitin-binding peritrophin (248)and trypsin (98) domains; while the fat body sequences showed high occurrence of cytochrome P450s (85) and protein kinase (123) domains. Further, the midgut transcriptome of A. planipennis revealed putative microbial transcripts encoding for cell-wall degrading enzymes such as polygalacturonases and endoglucanases. A significant number of SNPs (137 in midgut and 347 in fat body) and microsatellite loci (317 in midgut and 571 in fat body) were predicted in the A. planipennis transcripts. An initial assessment of cytochrome P450s belonging to various CYP clades revealed distinct expression patterns at the tissue level. CONCLUSIONS AND SIGNIFICANCE: To our knowledge this study is one of the first to illuminate tissue-specific gene expression in an invasive insect of high ecological and economic consequence. These findings will lay the foundation for future gene expression and functional studies in A. planipennis

An insertional mutagenesis programme with an enhancer trap for the identification and tagging of genes involved in abiotic stress tolerance in the tomato wild-related species Solanum pennellii

Salinity and drought have a huge impact on agriculture since there are few areas free of these abiotic stresses and the problem continues to increase. In tomato, the most important horticultural crop worldwide, there are accessions of wild-related species with a high degree of tolerance to salinity and drought. Thus, the finding of insertional mutants with other tolerance levels could lead to the identification and tagging of key genes responsible for abiotic stress tolerance. To this end, we are performing an insertional mutagenesis programme with an enhancer trap in the tomato wild-related species Solanum pennellii. First, we developed an efficient transformation method which has allowed us to generate more than 2,000 T-DNA lines. Next, the collection of S. pennelli T0 lines has been screened in saline or drought conditions and several presumptive mutants have been selected for their salt and drought sensitivity. Moreover, T-DNA lines with expression of the reporter uidA gene in specific organs, such as vascular bundles, trichomes and stomata, which may play key roles in processes related to abiotic stress tolerance, have been identified. Finally, the growth of T-DNA lines in control conditions allowed us the identification of different development mutants. Taking into account that progenies from the lines are being obtained and that the collection of T-DNA lines is going to enlarge progressively due to the high transformation efficiency achieved, there are great possibilities for identifying key genes involved in different tolerance mechanisms to salinity and drought

Transcriptomic Signatures of Ash (Fraxinus spp.) Phloem

Ash (Fraxinus spp.) is a dominant tree species throughout urban and forested landscapes of North America (NA). The rapid invasion of NA by emerald ash borer (Agrilus planipennis), a wood-boring beetle endemic to Eastern Asia, has resulted in the death of millions of ash trees and threatens billions more. Larvae feed primarily on phloem tissue, which girdles and kills the tree. While NA ash species including black (F. nigra), green (F. pennsylvannica) and white (F. americana) are highly susceptible, the Asian species Manchurian ash (F. mandshurica) is resistant to A. planipennis perhaps due to their co-evolutionary history. Little is known about the molecular genetics of ash. Hence, we undertook a functional genomics approach to identify the repertoire of genes expressed in ash phloem.Using 454 pyrosequencing we obtained 58,673 high quality ash sequences from pooled phloem samples of green, white, black, blue and Manchurian ash. Intriguingly, 45% of the deduced proteins were not significantly similar to any sequences in the GenBank non-redundant database. KEGG analysis of the ash sequences revealed a high occurrence of defense related genes. Expression analysis of early regulators potentially involved in plant defense (i.e. transcription factors, calcium dependent protein kinases and a lipoxygenase 3) revealed higher mRNA levels in resistant ash compared to susceptible ash species. Lastly, we predicted a total of 1,272 single nucleotide polymorphisms and 980 microsatellite loci, among which seven microsatellite loci showed polymorphism between different ash species.The current transcriptomic data provide an invaluable resource for understanding the genetic make-up of ash phloem, the target tissue of A. planipennis. These data along with future functional studies could lead to the identification/characterization of defense genes involved in resistance of ash to A. planipennis, and in future ash breeding programs for marker development

Transcriptomics of the Bed Bug (Cimex lectularius)

BACKGROUND: Bed bugs (Cimex lectularius) are blood-feeding insects poised to become one of the major pests in households throughout the United States. Resistance of C. lectularius to insecticides/pesticides is one factor thought to be involved in its sudden resurgence. Despite its high-impact status, scant knowledge exists at the genomic level for C. lectularius. Hence, we subjected the C. lectularius transcriptome to 454 pyrosequencing in order to identify potential genes involved in pesticide resistance. METHODOLOGY AND PRINCIPAL FINDINGS: Using 454 pyrosequencing, we obtained a total of 216,419 reads with 79,596,412 bp, which were assembled into 35,646 expressed sequence tags (3902 contigs and 31744 singletons). Nearly 85.9% of the C. lectularius sequences showed similarity to insect sequences, but 44.8% of the deduced proteins of C. lectularius did not show similarity with sequences in the GenBank non-redundant database. KEGG analysis revealed putative members of several detoxification pathways involved in pesticide resistance. Lamprin domains, Protein Kinase domains, Protein Tyrosine Kinase domains and cytochrome P450 domains were among the top Pfam domains predicted for the C. lectularius sequences. An initial assessment of putative defense genes, including a cytochrome P450 and a glutathione-S-transferase (GST), revealed high transcript levels for the cytochrome P450 (CYP9) in pesticide-exposed versus pesticide-susceptible C. lectularius populations. A significant number of single nucleotide polymorphisms (296) and microsatellite loci (370) were predicted in the C. lectularius sequences. Furthermore, 59 putative sequences of Wolbachia were retrieved from the database. CONCLUSIONS: To our knowledge this is the first study to elucidate the genetic makeup of C. lectularius. This pyrosequencing effort provides clues to the identification of potential detoxification genes involved in pesticide resistance of C. lectularius and lays the foundation for future functional genomics studies

New Introductions, Spread of Existing Matrilines, and High Rates of Pyrethroid Resistance Result in Chronic Infestations of Bed Bugs (Cimex lectularius L.) in Lower-Income Housing

Infestations of the common bed bug (Cimex lectularius L.) have increased substantially in the United States in the past 10-15 years. The housing authority in Harrisonburg, Virginia, conducts heat-treatments after bed bugs are detected in a lower-income housing complex, by treating each infested unit at 60°C for 4-6 hours. However, a high frequency of recurrent infestations called into question the efficacy of this strategy. Genetic analysis using Bayesian clustering of polymorphic microsatellite loci from 123 bed bugs collected from 23 units from May 2012 to April 2013 in one building indicated that (a) 16/21 (73%) infestations were genetically similar, suggesting ineffective heat-treatments or reintroductions from within the building or from a common external source, followed by local spread of existing populations; and (b) up to 5 of the infestations represented new genotypes, indicating that 5 new populations were introduced into this building in one year, assuming they were not missed in earlier screens. There was little to no gene flow among the 8 genetic clusters identified in the building. Bed bugs in the U.S. often possess one or both point mutations in the voltage-gated sodium channel, termed knockdown resistance (kdr), from valine to leucine (V419L) and leucine to isoleucine (L925I) that confer target-site resistance against pyrethroid insecticides. We found that 48/121 (40%) bed bugs were homozygous for both kdr mutations (L419/I925), and a further 59% possessed at least one of the kdr mutations. We conclude that ineffective heat treatments, new introductions, reintroductions and local spread, and an exceptionally high frequency of pyrethroid resistance are responsible for chronic infestations in lower-income housing. Because heat treatments fail to protect from reintroductions, and pesticide use has not decreased the frequency of infestations, preventing new introductions and early detection are the most effective strategies to avoid bed bug infestations in multistory apartment buildings